Document made available under the 
Patent Cooperation Treaty (PCT) 



International application number: PCT/IL04/001 181 

International filing date: 29 December 2004 (29.12.2004) 

Document type: Certified copy of priority document 

Document details: Country/Office: US 

Number: 60/533,263 

Filing date: 31 December 2003 (31.12.2003) 



Date of receipt at the International Bureau: 04 February 2005 (04.02.2005) 



Remark: Priority document submitted or transmitted to the International Bureau in 
compliance with Rule 17.1(a) or (b) 




World Intellectual Property Organization (WIPO) - Geneva, Switzerland 
Organisation Mondiale de la Propriete Intellectuelle (OMPI) - Geneve, Suisse 



PCT/IL20 04/001181 
2 h JAN 2005 



MM: 



WM 



PA 1269717 



3mm 



^ w fi^ ygi TnrwnCTt^^ m turn v 

— - ••■ 

, iTC>j Ai:ii, 'ii<))>yii<.> J >ifi; mi:s i:i ium:si:mits. shams. c omcEm 

UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

January 13, 2005 

THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COPY FROM 
THE RECORDS OF THE UNITED STATES PATENT AND TRADEMARK 
OFFICE OF THOSE PAPERS OF THE BELOW IDENTEBTED PATENT 
APPLICATION THAT MET THE REQUIREMENTS TO BE GRANTED A 
FILING DATE UNDER 35 USC 111. 

APPLICATION NUMBER: 60/533,263 
FILING DATE: December 31, 2003 



By Authority of the 
jW^ 1 I , COMMISSIONER OF PATENTS AND TRADEMARKS 

•-•'.Us 



P. SWAIN 
Certifying Officer 



-vtJ!.. 



-X 



CO 



CDs 



§1 



Co 



Patent and Traaemark Office; US: blEPARl^ENf OF "COMMERCI ' 
Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of Information unless it displays a 
valid OMB control number. . • >V % **• :*• " 

PROVISIONAL APPLICATION FOR PATENT COVER SHEET 

This is a request for filing a PROVISIONAL APPLICATION FOR PATENT under 37 CFR 1 .53(c). 



Given Name (first and middle [if any]) 



Gavriel 
Hagai 



INVENTOR(S) 



Family Nam or Surname 



MERON 
KRUPNIK 



Residence 
(City and either State or Foreign Country) 



Petach Tikva, Israel 
Nofit, Israel 



□ Additional inventors are being named on the A separately numbered sheets attached hereto 



TITLE OF THE INVENTION (280 characters max) 



SYSTEM AND METHOD FOR DISPLAYING AN IMAGE STREAM 



Direct all correspondence to: 
Customer Number 

OR 



CORRESPONDENCE ADDRESS 





27130 



Type Customer Number here 



j§] Firm or 

• Individual Name 



Address 10 Rockefeller Plaza 



Address 



City 



Country 



Place Customer Number 
Bar Code Label here 



Eitan, Pearl, Latzer & Cohen Zedek, LLP. 



Suite 1001 



New York 



USA 



State 



Telephone 



New York 



212-632-3480 



ZIP 



10020 



Fax 212-632-3489 



ENCLOSED APPLICATION PARTS (check all that apply) 



Specification Number of Pages I 22 I □ CD(s), Number , 



I3 Drawing(s) Number of Sheets | af 



□ Application Data Sheet. See 37 CFR 1 .76 |g] Other (specify) 



postcard 



METHOD OF PAYMENT OF FILING FEES FOR THIS PROVISIONAL APPLICATION FOR PATENT (check one) 



□ Applicant claims small entity status. See 37 CFR 1 .27. 

□ A check or money order is enclosed to cover the filing fees 

£>3 The Commissioner is hereby-authorized to charge filing fees 

_^ or credit any overpayment to Deposit Account Number • 

□ Payment by credit card. Form PTO-2038 Is attached. 



06-0649 



FILING FEE 
AMOUNT ($) 



160 



The invention was made by an agency of the United States Government or under a contract with an agency of the 
United States Government 
03 No. 

□ Yes, the name of the U.S. Government agency and the Government contract number are: 



!« 

co cm 

3 CO 
CO 

h-— * •& 

con 



so 



ICO 




Date 



31 / Dec / 2003 



SIGNATURE 

TYPED or PRINTED NAM 
TELEPHONE 



212-632-3480 



REGISTRATION NO. 

(if appropriate) 



Docket Number 



37,912 



P-6472-USP 



USE ONLY FOR FILING A PROVISIONAL APPLICATION FOR PATENT 

This collection of information is required by 37 CFR 1.51. The information is used by the public to file (and by the PTO to process) a 
provisional application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 8 hours to 
complete, including gathering, preparing, and submitting the complete provisional application to the PTO. Time will vary dep nding upon the 
individual case. Any comments on th amount of time you require to complete this form and/or suggestions for reducing this burd n, should 
b sent to the Chief Information Offic r t U.S. Patent and Trademark Office, U.S. Department of Commerce, Washington, D.C. 20231. DO 
NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Provisional Application, Commissioner for Patents, P.O. Box 
1450, Alexandria, VA 22313-1450. 



Copy provided by USPTO from the IFW Im^age Database on 01/06/2005 




• \V* United States Pr visi nal Patent Applicati nFn 

SYSTEM METHOD FOR DISPI^ 

5 FIELD QF THE INVENTION 

. The present invention relates, to a method and system for displaying and/or 
reviewing. image streams. . 

BACKGROUND OF THE INVENTION 
An image stream may be assembled from ^series of still images and 

10 displayed to. a user. The images may be created or collected from various 
sources. For example, U.S. Patent No. 5,604,531 to Jddan et al., assigned to the 
common assignee of the. present/application and incorporated herein by reference, 
teaches, inter alia, an in-vivo imaging system which in one embodiment includes a 
swallowable capsule. The imaging system may, for example, capture images of a 

is lumen such as the gastrointestinal (Gl) tract and may transmit them to an external 
. recording device. Large numbers of images may be collected for viewing and, for 
example, combined in sequence. An image stream of, for example, 40 minutes in 
length, containing for example about 4,800 frames, may be available to, for 
example to a health professional for review. Other numbers of frames or lengths 

20 may be used. . In one embodiment, a health professional may, for example, use the . 
images to diagnose pathological conditions of the Gl tract,. and, in addition, the 
%stem may provide information about the location of these pathologies. 
Pathology, if present may typically be found in only a few locations along the long 
traverse of the Gl tract, and therefore in a small percentage of the recorded 

25 images. ' ' : ' * 
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• ;•..',* "In general, a user may try to set the streaming rate to the highest rate 
. where the user can quickly review the image stream without missing important 
.• information thatmay.be present in any of the images included in the stream. The 
rate at which a .user can effectively review an image stream may be limited by a 
5 physiological averaging above which certain details in individual images displayed 
in the stream may be physiolpgirally .filtered out. 

' ; SUMMARY OF THE INVENTION 

. The system and method according to embodiments of the present invention may 
10.'. allow an image stream to be. viewed while emphasizing images that may be of 
interest (e.g., images of pathologies). Thus, a user may quickly locate selected 
images of interest out of a lengthy image stream gnd an efficient and short view 
time may be, provided. 

In one embpdiment, a system and method method may display an in vivo 
15 image stream. An in vivo image stream may be displayed as a multi-frame stream, 
and frames displayed in a single time slot may be spatially positioned based on for 
example a predetermined criterion. 

In one embodiment, a system and method are provided for displaying an 
image stream, such that multiple images from the image stream are displayed 
20 substantially simultaneously in a time slot. As such a single-frame stream may be 
'converted to a multi-frame stream that may be shorter in length. In a further 
embodiment of the invention, the spatial position of the images displayed 
substantially . simultaneously in each time slot may be adjusted based on a 
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v predetermined criterion. For example, the spatial order of the images displayed 
' substantially simultaneously in each time slot may correspond to a degree of 
• variation of each image compared to a selected or generated reference image. In a 
: /further example, images displaying pathological looking tissue may be positioned 
5 : away from images displaying healthy looking tissue. In one embodiment of the 
. .invention, images from a group of images displayed substantially simultaneously, 
\ .'containing a high degree of variation as compared to the other images in the group 
may be displayed in a spatial position where a viewer may typically draw his/her 
eyes first. In certain embodiments, the images may be collected from a 
10 .'swallowable device, such as a. capsule or other suitable shaped device, traversing 
the Gi tract. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood and appreciated more fully from 
15 the following detailed description taken in conjunction with the drawings in which: 

. Figure 1A shows a schematic diagram of an in-vivo imaging system 
according to one embodiment of the present invention; 

Figure 1B shows a schematic diagram of a monitor, display according to one 
embodiment of the present invention; 

e 

20 Figure 2 shows a portion of a display according to an embodiment of the 

present invention; 

3 . 
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Figure 3 shows a flow chart describing a method for spatially positioning a 
plurality of image frames displayed substantially simultaneously, according to an 
, embodiment of the present invention; 

Figure 4 shows a flow chart describing a method for spatially positioning a 
5 plurality of image frames displayed substantially simultaneously, according to 
another embodiment of the present invention; 

Figure 5 shows a portion of a display according to a second embodiment of 
the present invention; 

Figure 6 shows an exemplary flow chart describing a method for combining 
.10 multiple image streams into a single image stream according to an embodiment of 
the present invention; and 

■ • . • Figure 7 showing an exemplary flow chart for adjusting the stream rate of 
the multi-frame stream according to one embodiment of the present invention, 

15 DETAILED DESCRIPTION OF THE INVENTION 

In the following description, various aspects of the present invention will be 
described. For purposes of explanation, specific configurations and details are set 
forth in order to provide a thorough understanding of the present invention. 
However, it will also be apparent to one skilled in the art that the present invention 
20 may be practiced without the specific details presented herein. Furthermore, well- 
•Rnown features may be omitted or simplified in order not to obscure the present 
invention. . 
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Reference is made to Fig. 1A, which shows a schematic diagram of an in- 
vivo imaging system according to one embodiment of the present invention. In an. 
exemplary embodiment, the system may comprise an imaging device, such as for 
example a device 40 which may, for example, be capsule shaped, having an image 

• 5 sensor 46, for capturing images, an illumination source 42, for illuminating the body 
lumen, a transmitter 41 , and an antenna 44, for transmitting images and possibly 
other information to, for example, a receiving device. Typically, the imaging device 
may correspond to embodiments described in U.S. Patent No. 5,604,531 to Iddan 
et al.„ and/or to embodiments described in published application WO01/65995 to 

10 Glukhovsky et al. a both of which are incorporated herein by reference in their 
entirety, but in alternate embodiments there may be other types of imaging 
devices. Device 40 may be a capsule, but may have other suitable shapes or 
configurations. In one embodiment, an imaging device may include more than one 
image sensor. The image sensors may, for example be arranged at either end of 

15 device 40, or at the same end of the capsule, in slightly different positions or 
different angles. A device, which includes a plurality of image sensors, may be. 
described for example, in International Publication Number WO 02/054932 which 
is assigned to the common assignee of the present invention and which is hereby 
incorporated by reference. 

20 Reference is now made to Fig. 6, showing an exemplary flow chart 

•describing a method for combining multiple image streams into a single image 
stream according to one embodiment of the present invention. Each image sensor 
may capture in vivo images and transmit the in vivo images via the transmitter. 41 
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or via separate transmitters. The separate. image streams obtained from the 
multiple image sensors may be stored (step 610). . Comparison may be made 
between frames captured from the separate image streams (step 620), In other 
embodiments of the invention, the separate image streams may not be stored in 

5 their entirety or at all before comparison, is made. Based on the comparison, 
frames from each of the separate image streams may be slotted or placed into a 
single image stream (step 630). Other operations and combinations of operations 
may be used. Other methods besides or in addition to comparing frames may be 
used to combine multiple image streams. The separate image streams may be 

10 combined such that frames displaying similar scenery are slotted or placed in close 
proximity to each other. Other suitable criteria may be used as well. 

In another embodiment of the invention, device 40 may include sensors 43 
other than image sensors such as for example, a temperature sensor, a pressure 
sensor, a blood sensor, a pH sensor, an ultrasound sensor, an oxygen sensor, 

is etc., and may include more than one sensor. Data sampled from sensor 43 may 
for example be transmitted via transmitter 41 or via separate transmitters, or need 
not be transmitted. 

Typically, located outside the patient's body in one or more locations may 
be an image receiver 12, a data processor 14, and an image monitor 18. image 
20 receiver 12 may typically include an antenna or antenna array (not shown) and an 
rimage receiver storage unit 16. Data processor 14 may include a processor 19 
and a storage unit 21 . In one embodiment of the present invention, storage unit 21 
may include an image database 22. Image monitor 18 may display, inter alia, 
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images recorded by,, for example, device 40. Typically, data processor 14 and 
monitor 18 may be part of a personal computer or workstation, which includes 
standard components such as processor 19, a memory, a disk drive, and input- 
output devices, although alternate configurations are possible. Data processor 14 

5 ■ may typically, as part of its functionality, act as a controller controlling the display of 
the images. Image monitor 18 may typically be a conventional video display, but 
may, in addition, be any crther device capable of providing images or other data 
• and may be of any size monitor including large projection size monitors. . The 
image monitor 18 presents the image data, typically in the form of still and 

10 streaming image frames, and, in addition may present other, information. In an 
exemplary embodiment, the various categories of information are displayed in 
windows. Other displaying formats may be used. Multiple monitors may be used 
to display image and other data. The images captured by the imaging system may 
be of any suitable shape including for example circular, square,, rectangular, or 

15 hexagonal, oval etc. 

In operation, image sensor 46 may capture images and may send data 
representing the images to transmitter 41, which may transmit data to image 
receiver 12 using, for example, electromagnetic radio waves. Image receiver 12 
may transfer the image data to image receiver storage unit 16. After a certain 

20 period of time of data collection, the image data stored in storage unit 16 may be 
.transferred to the data processor 14 or the data processor storage unit 21 ■ For 
example, the image receiver 12 or image receiver storage unit 16 may be taken off 
the patient's body and may be connected to the personal computer or workstation 
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that may include the data processor 14 via a standard data link, e.g., a serial, 
• * * » 
parallel, USB, or wireless interface of known construction. According to one 

■embodiment the image data may then be transferred from the image receiver 

storage unit 16 to an image database 22 within data processor storage unit 21. 

typically, the image stream is stored as a series of images in the image database 

22,. which may be implemented in a variety of known manners. Data processor 14, 

including possibly dedicated software may analyze the data and provide the 

analyzed data to the image monitor 18, where a user views the image data. In one 

embodiment of the invention, more than one image stream may be captured from 

. rnore.than one image sensor. In4his embodiment, data processor 14 may be used 

to combine the separate image streams into one image stream. The combining of 

.the two or more image streams may be for example, based on comparing images 

from each image stream, based on time of image capture, or "based on other 

.. criteria or more than one criterion. Other configurations allow for real time viewing 

where, for example, device 40 or imager receiver 12 may transmit data in real time 

to data processor 14 for real time viewing on monitor 1 8. 

/;. Typically, according to one embodiment, an in-vivo imaging device, such as 

for example device 40, may collect a series of still images as it traverses the Gl 

tract. In one embodiment of the present invention, device 40 may collect a large 

volume of data, as it may. take several hours to traverse the Gl tract, and may 

;record images at a rate of, for example, two images every second, resulting in the 

recordation of, for example, thousands of images. The image recordation rate (or 

frame capture rate) and recordation duration may be varied. 
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Typically, the image data captured and transmitted by the device 40 may be 
digital color image data, although in alternate embodiments other image formats 
may be used. In an exemplary embodiment, each frame of image data may 
include 256 rows of 256 pixels, each pixel may include bytes for color and 

' 5 brightness, according to known methods. In one embodiment, 4x4 "sub" pixels 
may be included, each sub-pixel including red, green and blue data, one color per 
• pixel, with one color repeated. Other suitable methods of recording image data 
may be used. . Typically, images may be stored sequentially in data processor 
storage unit 21 . dther image formats may be used. 

10 • • Typically, data processor storage unit 21 may store a series of images 
captured by device 40, for example, as it moves through a patient's Gl tract (other 
. lumens may be imaged). The series of images may be combined, typically 
consecutively, to form an image stream. When viewing the image stream, the user 
may be typically presented with one or more windows on monitor 18; in alternate 

15 embodiments multiple windows need not be used and only the image stream may 
be displayed. Reference is now made to Fig. 1B showing a schematic diagram of 
display 18 according to one embodiment of the invention. In an embodiment 
where multiple windows are provided, for example, an image window 50 may 
display the image stream or portions (for example still portions) of that image 

20 stream. A sensor window 53, may display a concurrent stream of data from 
sensors other than image sensors, for example, ultrasound, pressure, temperature, 
blood, oxygen, and/or a pH sensor. Other sensors may be used. A location 
Window 52 may display information on the location of the imaging device, for 

. 9 
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example, device 40, at the time of capturing the image frames being displayed. A 
control window 54 may include buttons or other controls that may alter the display 
of the image; for example, stop, play, pause, capture image, step, fast-forward, 
rewind, or other. Such controls may be activated by, for example, a pointing device 

5 such as a mouse or trackball. Typically, the image stream may be frozen to view 
one frame or one set of frames, speeded up, or reversed; sections may be 
skipped; or any other method for viewing an image may be applied to the image 
stream. Other windows may be included. 

While the following discussion relates to the case where data from a device 

10 40 is stored for later use, the system and method of the present invention may be 
used with systems allowing for real time viewing of image data. 

. Furthermore, while typically* the components accepting, processing and 
displaying the image data are contained within a workstation system or PC, other 
systems may be used, and other (e.g., distributed) components may perform such 

is image accepting, processing and displaying. For example, a large projection type 
monitor may be used to view a plurality of images substantially simultaneously. 

In one embodiment, the imaging window may display a shortened multi- 
frame stream where multiple images are displayed substantially simultaneously in 
each time slot. For example, in one embodiment, the original image stream may 

20 be divided into, for example, separate image streams displayed substantially 
simultaneously in single time slots. Various suitable methods may be used to 
separate an original image stream into a plurality of image streams to be streamed 
substantially simultaneously. In one example, images from the original image 
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stream may be divided into, for example, 16 subset image streams such that, for 
example, during the first time slot, the first 16 images of the original image stream 
may be displayed substantially simultaneously, and in the second time slot the next 
16 images of the original images stream may be. displayed substantially 

5 simultaneously, etc. Other number of image streams may be used and some 
frames in the original stream may be skipped. In. alternate embodiments, the 
images may be separated for example, placed in different files or memory blocks in 
processor 14. In one embodiment, each resulting image stream includes, a 
separate subset of images from the original image stream; in alternate 

10 embodiments, the images from each resulting image stream may overlap. In 
another embodiment of the invention, images of the original image stream may be 
divided between the separate image streams in an alternate manner; 

In one embodiment, images from the original image stream may be directed 
to the proper screen position at viewing time, and the image stream may not 

15 actually be separated. In such embodiments, a different number of images may be 
displayed substantially simultaneously for every time slot. In one embodiment, 
each time slot may display all image frames captured from a specified region. For 
example, the first time slot may include all frames captured in the esophagus, the 
second time slot may display all images captured in the stomach, etc. In an 

20 alternate embodiment the same number of image frames may be displayed in 
each time frame. 

Fig. 2 depicts a. portion of a display according to an embodiment of the 
present invention. Referring to Fig. 2, the display 200 includes, as an example. 16 
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image frames displayed substantially simultaneously in a single time slot (21 0A- 
240D). In an exemplary embodiment, the windows or viewing areas, 210A, 210B 
etc., are close together, with a minimum of blank or black space between the 
frames, and typically are displayed in matrix form, to allow a viewer to see the 

5 entirety of the frames without substantially moving his eyes. Other suitable spatial . 
arrangements may be used, 

The spatial order of the image frames displayed in a single time slot may be 
adjusted. The spatial order of the image frames displayed substantially 
simultaneously may correspond to the chronological order that the images were 

10 captured, may correspond to an alternate criterion, or to more than one criterion. 
In one embodiment of the invention the spatial orider of the frames displayed may 
be based on a specified criterion of interest. In one embodiment of the invention, 
the image frames displayed in a single time slot are spatially positioned based on a 
criterion of color variation. 

15 A set of operations corresponding to an exemplary algorithm, or method is 

listed in Fig. 3, according to one embodiment of the invention. Referring to block 
318, a set of image frames to.be displayed substantially simultaneously in a single 
time slot may be selected. Normalization of lighting or other normalization may be 
performed on the image frames to be displayed (block 320). This may be 

20 performed to, for example, reduce any color variation due to shadowing, variation 
in lighting intensity, or variation in lighting and image sensor properties when image 
frames may be captured from more than one image sensor. Subsequently, a 
reference image may be generated b^sed on the image frames to be displayed 
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• (330).- In one embodiment of the invention, the reference image frame may be, for 
example, an image representing the average or sum of the image frames to be 
. displayed substantially simultaneously in a single time slot. In another embodiment 
* of. the present invention, the reference image may be, for example, a selected 

5 Image frame that may . have some -known properties, an image selected or 
generated from a previous time slot, or other suitable images. In one embodiment 
of the invention, the reference image may represent a typically healthy looking 
tissue. In step 340, the degree of variation of frames from reference image may be 
determined. The variation may be based on comparing sections of images or may 

10 be based on . comparing whole images. In one example, images displaying 
pathology may show a high degree of variation with respect to the reference image, 
for example when the reference image may represent a typically healthy tissue. A 
score may be given to each image, based on the comparison performed (block 
350). In another embodiment of the invention, image frames to be displayed in a 

15 single time slot may be. compared to more than one reference image and receive 
more than one score. The spatial position of the image frames displayed may be 
based on the scores they were assigned (360). For example, according to one 
embodiment pictures having similar scores may be positioned where a viewer may 
typically draw his/her eyes first, e.g., at one comer of the display. Other suitable 

20 operations or sets of operations may be performed. 

Other post processing methods may be included as well to differentiate 
between images that may display, for example, pathology and images that may 
display, for example, bubbles, content, or other variations through the Gl tract, 
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iwhich may not be associated with pathology or other criteria of interest, in another 
embodiment of the present invention, the generated image may represent an 
average image generated from the entire original image stream, from a 
predetermined section of the original image stream, or from image streams taken 
. from a pool of volunteers. In another example the generated and/or pre-selected 
image may represent a pathology or specific information that is being sought in the 
.plurality of image streams. As such, images with the lowest degree of variation 
with respect to the reference image may be representative of the images of 
interest. Other suitable algorithms may be used as well. As an example, images 
detected as displaying blood may be a criterion of interest. Algorithms for blood 
. detection are described, for example, in embodiments described in US Application 
Number 10/097,096 filed on March 14, 2002, assigned to the common assignee of 
the present invention and hereby incorporated by reference. In another 
embodiment of the invention other criteria besides or together with color variation 
may be used. For example, shape variation or texture variation may be used as a 
criterion to help identify, for example, polyps in the Gl tract. Variation above or 
below a certain threshold based on a selected or generated image may be used. 
. The threshold may be adjusted to increase or decrease sensitivity. Other suitable 
criteria may be used as well. 

In yet another embodiment of the invention, the criteria of interest that 
determines the spatial order of the image frames in each time slot, and the number 
of images displayed per time slot may be based on readings concurrently obtained 
from sensors other than image sensor, for example the spatial positioning of image 
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frames displayed substantially simultaneously in a single time slot may be based 
on reading obtained from one or more non-image sensors such as the sensors 
mentioned above. Reference is now made to Fig. 4 showing a flow chart for 
sorting image frames displayed substantially simultaneously in a single time slot, 

5 according to one embodiment of the invention. In step 510, images to be 
displayed substantially simultaneously may be selected. Corresponding sensor 
readings from one or more non-image sensors (e.g. temperature sensor, pressure 
sensor, blood sensor, pH sensor, ultrasound sensor, oxygen sensor, etc.) may be 
selected (step 520). For example, temperature readings recorded during the time 

10 of capturing the image frames selected in step 51 0 may be used. Images selected 
in step 510 may be sorted (step 530) based on the corresponding sensor data 
selected in step 520, in for example ascending or descending order. Other 
methods of sorting may be implemented. In one embodiment of the present 
invention, sorting may be based on reading of more than one sensor and may be 

15 based on image analysis as well as sensor readings. In another embodiment of 
■ the present invention, images may also be sorted based on a one or zero output of 
a sensor, so that all images corresponding to a one output may be grouped 
together and all images corresponding to a zero output may be grouped together. 
Images corresponding to sensor readings passing a defined threshold may be 

20 tagged (540). Such images may be emphasized during display, for example, as is 
described below. Other operations or series of operations may be used. 

typically, increasing the number of image frames displayed substantially 
simultaneously in a single time slot, may allow for the . processing method 

• .15 • . ' • 
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described, for example, by way of example, in Fig. 3 and Fig. 4 to single out, for 
example, a relatively small number of frames showing pathology from, for example, 
a relatively large number of frames showing typically healthy looking tissue. . 
Increasing the number of images displayed in a single time slot decreases the 

5 . emphasis, on small variation in, for example, healthy, tissue while increasing the 
emphasis on large variations that appear in for example, pathological tissues. In 
addition, according to one embodiment, increasing the number of images 
displayed in a single time slot decreases the length of the displayed image stream. 
In one embodiment of the invention, image frames that showed, for 

10 example, a high degree of variation may be positioned in a location where the user 
is mostly likely to glance first, and image frames that showed, for example, a low 
degree of variation may be positioned where the user is most likely to glance last. 
In one embodiment of the present invention, it may be assumed that the user will 
scan the matrix of images starting at the upper left hand corner. In this 

is embodiment, image frames that may be of interest, for example showing a high 
degree of variation with respect to frames, for example, representing healthy 
looking tissue, may be positioned in the upper left hand corner, starting with for 
example frame 21 OA (Fig. 2). Image frames that may not be of interest, for 
example, images showing a low degree of variation with respect to images 

20 representing healthy looking tissue, may be positioned in the lower right hand 
corner, ending with for example frame 240D (Fig. 2). Frames may be arranged by 
descending order of degree of variation. The progression from frame 21 OA to 
frame 240D may be by row based order (e.g. 21 OA, 21 OB, ...220A, 220B, ...230A, 

■ . : 16 . 
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..240D), by column based order (e.g: 210A, 220A, ..i 210B, 220B, ...210C, ' 
...240D), or by radial based order (e.g. 21 OA, 210B, 220B, 220A, 210C, 22QC, 
230C, 230B, ...240D). Other suitable progression series may, be used, such as 
spiral based order. In another embodiment of the present invention, image frames 

5 that may be of interest, for example showing a high degree, of variation, may be • 
positioned in the center of the image window, at a height of for example two thirds 
(2/3) above the bottom, while images showing a low degree of variation may be 
positioned in the periphery of the image window. The progression from image 
frames showing a high degree of variation to image frames showing a low degree 
. 10 of variation may be a spiral shaped. Spiral progressions may be clockwise spiral . 
progressions, or counter-clockwise progressions. Other suitable criteria may be 
used besides degree of variation, and other suitable, ranges may be used. 

Other suitable image frame arrangements may be used as well. For 
example, image frames in single time slot that meet a particular criterion of interest 

is above a defined threshold may in one embodiment of the. present invention, be 
grouped together for emphasis, by for example, fusion at the borders to form a 
single entity, or emphasized by, for example, highlighting, inserting a border around 
grouped image frames, etc. For example, in Fig. 2 image frames 21 OA and 21 OB 
are highlighted by border 299. Other number of frames or types of grouping may 

20 be used. 

In one embodiment of the invention, images displayed substantially 
simultaneously may be of different size or shape. In one example, specific images 
from the group of images displayed substantially simultaneously may be enlarged 
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.or displayed in a larger size so as to emphasize them. Reference is now made to 
Fig, 5 showing 46 image frames displayed substantially simultaneously, according 
to one embodiment. In one embodiment of the invention, image frames having for 
example, the highest degree of variation according to a specified criterion are place 
5 . in for example ..the center, at a height of for. example two thirds (2/3) above the 
bottom. In one example, images with variation above a predetermined threshold 
may be grouped together and. displayed in larger frames such as frames 401, 402, 
. 403, 404, and 405, 406. Other images below the specified threshold, for example, 
might be displayed in smaller frames such as, for example, frames 407 to 446. 
10 The sensitivity of this grouping may be adjusted by increasing or decreasing the 
defined threshold. The threshold may be adjusted by the user or by an algorithm 
. adjusting the threshold based on a defined maximum and minimum number of 
large frames to be displayed concurrently, in a single time slot along* with other 
criteria. In another embodiment of the invention, the larger frames may be 
15 positioned at the top left corner. In yet another embodiment of the present 
invention different size frames may be positioned in other suitable arrangement. 
As such, a large number of images, may be reviewed by a u?er, such as a health 
professional, while only emphasizing the typically few images most likely to contain 
information of interest. The typically large majority of images less likely to contain 
20 . information of interest may be reviewed at a quick glance. In one embodiment of 
the invention, a user may pause the image streaming when identifying an image of 
particular interest. As such the user may examine more carefully the image or 

images of interest and the images taken around the vicinity of the image of interest 

*♦ ... . • . 
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/ V : Jhe order that the images are shown substantially simultaneously may be altered 
, to show images for example in a chronological order. Other suitable features may 
be used as well to aid in the analysis of the images obtained. 

! In one embodiment of the invention, lighter images may be in general 
5.. positioned in the upper part of the image window 50 (Fig. 1B), while in general 
s darker images may be positioned in the lower end of image window 50. As such 
;. the continuity between subsequent time slots may be enhanced for easier viewing 
of a stream. : - ■ ' 

Typically, if normally a single-frame stream is displayed at a certain rate, a 
io . multi-frame stream may be displayed at a slower rate. For example if typically a 
. single-frame stream is displayed at a rate of 20 frames per second, a multi-frame 
• • stream may be displayed for example at a rate of for example 5 or less frames per 
: second. In such a case the overall viewing time may be up to four times less than 
. the original single-frame stream viewing time. Other suitable viewing rates and 
15 ratios between single-frame stream and multi-frame stream viewing time may be 
used. In one embodiment of the invention, the viewing rate of the multi-frame 
. stream may be adjusted based on the content of the frames displayed in each time 
slot and based on the transition of sequential time slots. Reference is made to Fig. 
7 showing an exemplary flow chart for adjusting the stream rate of the multi-frame 
20 stream according to one embodiment of the present invention. In one example, a 
set of frames to be displayed in a single time slot may be examined (step 710). If 
one or more frames. meet a predetermined criterion (step 720), such as a criterion 
of interest, the stream rate may be adjusted (step 750), for example, reduced. This 
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may help to further emphasize frames that may be of interest. In another example, 
.the, viewing rate may be reduced where there may be a large variation in the color 
content in the transition between neighboring time slots. Frames displayed in 
' neighboring time slots may be compared (step 730). If there is a large variation in 

5 frames, displayed in neighboring time slots, for example a large variation in color 
content or texture, the stream rate may be reduced (step 740). If there is no 
significant variation and no frames meeting a criterion of interest, the stream rate 
may be maintained (step 760). In yet another embodiment of the invention, a user 
may adjust the frame rate as well as the number of frames displayed substantially 
;io simultaneously. In an exemplary embodiment, the user may switch modes, 
• between viewing the images as one stream and viewing the images as a plurality 
of streams using a control such as a keystroke, mouse, or on-screen button. The 
user may control the plurality of streams in a manner similar to the control of a 
single stream, for example by using on screen controls. In an alternate 

is embodiment, only one mode may be offered to the user. In yet an alternate 
embodiment of the invention, a user may select a mode of only viewing images 
meeting a defined criterion of interest. 

Each image may be displayed with different post-processing. For example, 
one image may be subject to certain filtering or manipulation (e.g., red or green 

20 filtering, contrast enhancement, brightness alteration) and the other image may be 
subject to different or no filtering or manipulation. In one embodiment two or more 
images in a plurality of streams displayed substantially simultaneously are fused 
together at the borders and displayed as a single entity. As such, a user may 
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comfortably and concurrently incorporate information shown in each of the images 

while avoiding the distraction caused by the typically sharp contrast between 
• connecting edges or between the images and the background color which may 

appear between the images when the images are spaced apart. According to 
5 some embodiments fusing borders between independent images may be 

accomplished by, for example, one or more post processing algorithms known in 

the art, including but not limited to, smoothing convolution, mirroring, overlapping, 
: linear or non-linear fade-out fade-in, truncation, linear shape distortion, non-linear 

shape distortion, normalization or intensity, or other suitable post-processing. 
10 It will be appreciated by persons skilled in the art that the present invention 

is not limited by what has been particularly shown and described hereinabove. 

Rather the scope of the invention is defined by the claims that follow: 
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CLAIMS 

1 . A method for displaying an in vivo image stream according to the 

specification and drawings. 

2. A system for displaying an in vivo image stream according to the . 
5 . specification and drawings, . .. . ; s . ; V : , 



22 



Codv orovided by USPTO from the IFW Image Database on 01/06/2005 




Coov orovided bv USPTO from the IFW Image Database on 01/06/2005 



1 




SELECT IMAGE FRAMES TO BE 
DISPALYED 




r 


NORMALIZE LIGHTING 




r . 


GENERATE OR SELECT 
REFERENCE IMAGE 




r 


: DETERMINE DEGREE OF 
VARIATION OF FRAMES FROM 
REFERENCE IMAGE 




r 



318 



320 



330 



340 



SCORE 



J 



350 



SET SPATIAL POSITIONING 



360 



I 

V< 

c- 



t 
c 



Figure 3 



Coov provided bv USPTO from the IFW Image Database on 01/06/2005 




Copy provided by USPTO from the IFW Image Database on 01/06/2005 





• • ' ®kr$^ ' cf?&$^ ■ k -. , , .'- 

* - ,J\V - ^ - : i . j J'-;" ^ . V. , <- • ■ {,} j..y.. .. 



1* 



:'>:>;■!•:.<: 

ov"--: :i 

IS 



:,Viv\* 



Jpttf 



wts. 



mm 



SIS! 



S 

.i l, -.£'s, , f .r;W« - '»:.-.i*."! ••'r'5: . 



i 



Copy provided by USPTO from the IFW Image Database on 01/06/2005 



610 




COMPARE FRAMES FROM THE 
SEPARATE IMAGE STREAMS 



SLOT FRAMES FROM SEPARATE 
IMAGE STREAMS INTO A SINGLE 
IMAGE STREAM 



630 



fc. 

8 

a 
3 

I 
fc 



Figure 6 ; 



Copy provided by USPTO from the IFW Image Database on 01/06/2005 




IMMOBILIZING A HOUSING 
ON AN IN VIVO SURFACE 



-800 



FOCUSING AN IMAGING DEVICE 
ON A SITE TO BE IMAGED 



-802 



FIG. 8 



USPTO from the IFW Image Database on 01/06/2005 



